AIT 526 - Project¶
Sentiment Analysis of Amazon Online Purchase Ratings¶
Team 2¶
1. Fangxin Zhang
2. Muhammad Hassan
3. Shirinithi Thiruppathi
4. Tewodros Tamene
Dr. Duoduo Liao December 3, 2021
Note:¶
In this project notebook all the code that was developed for the final outcome is included. This notebook includes:
1. Libraries
2. Importing Data
3. Data Exploration
4. Data Preprocessing
5. Text Preprocessing
6. Exploratory Data Analysis
7. Popularity Based Recommender Analysis
8. Time-Series Analysis
9. TextBlob - Polarity
10. Text Analysis
11. Feature Engineering
12. TF-IDF
13. Sentiment Model Development
14. Model Selection
15. Hyperparameter Tuning
16. Classification Metrics
17. Code References 1. Libraries¶
In [1]:
import os
import json
import gzip
import pandas as pd
from urllib.request import urlopen
import random
import numpy as np
from tqdm import tqdm_notebook as tqdm
from collections import defaultdict
In [2]:
#Basic libraries
import pandas as pd
import numpy as np
#NLTK libraries
import nltk
import re
import string
#from wordcloud import WordCloud,STOPWORDS
from nltk.stem.porter import PorterStemmer
from sklearn.feature_extraction.text import TfidfVectorizer
# Machine Learning libraries
import sklearn
from sklearn.svm import SVC
from sklearn.preprocessing import LabelEncoder
from sklearn.preprocessing import StandardScaler
from sklearn.preprocessing import MinMaxScaler
from sklearn.ensemble import ExtraTreesClassifier
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import GridSearchCV
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.naive_bayes import BernoulliNB
from sklearn.neighbors import KNeighborsClassifier
from sklearn.multiclass import OneVsRestClassifier
from sklearn.svm import SVC
from sklearn.pipeline import Pipeline
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import label_binarize
from sklearn import svm, datasets
from sklearn import preprocessing
#Metrics libraries
from sklearn import metrics
from sklearn.metrics import classification_report
from sklearn.model_selection import cross_val_score
from sklearn.metrics import roc_auc_score
from sklearn.metrics import roc_curve, auc
#Visualization libraries
import matplotlib.pyplot as plt
from matplotlib import rcParams
import seaborn as sns
from textblob import TextBlob
from plotly import tools
import plotly.graph_objs as go
from plotly.offline import iplot
%matplotlib inline
#Ignore warnings
import warnings
warnings.filterwarnings('ignore')
#Other miscellaneous libraries
from scipy import interp
from itertools import cycle
#import cufflinks as cf
from collections import defaultdict
from collections import Counter
from imblearn.over_sampling import SMOTE
In [3]:
import cufflinks as cf
import plotly.offline
cf.go_offline()
cf.set_config_file(offline=False, world_readable=True)
2. Importing Data¶
In [4]:
import pandas as pd
import gzip
def parse(path):
g = gzip.open(path, 'rb')
for l in g:
yield json.loads(l)
def getDF(path):
i = 0
df = {}
for d in parse(path):
df[i] = d
i += 1
return pd.DataFrame.from_dict(df, orient='index')
df = getDF('Software.json.gz')
In [5]:
df.head()
Out[5]:
In [6]:
# Rating average
ratings = []
for review in parse("Software.json.gz"):
ratings.append(review['overall'])
print(sum(ratings) / len(ratings))
In [7]:
# Creating a copy dataframe
process_reviews =df.copy()
3. Data Exploration¶
In [8]:
# Shape of the data
process_reviews.shape
Out[8]:
In [9]:
# Check the datatypes
process_reviews.dtypes
Out[9]:
In [10]:
# Five point summary
process_reviews.describe()['overall'].T
Out[10]:
In [11]:
# Find the minimum and maximum ratings
print('Minimum rating is: %d' %(process_reviews.overall.min()))
print('Maximum rating is: %d' %(process_reviews.overall.max()))
4. Data Preprocessing¶
Handle missing data¶
In [12]:
# Checking for null values
process_reviews.isnull().sum()
Out[12]:
In [13]:
process_reviews['style']=process_reviews['style'].fillna('Missing')
In [14]:
process_reviews['reviewerName']=process_reviews['reviewerName'].fillna('Missing')
In [15]:
process_reviews['vote']=process_reviews['vote'].fillna('Missing')
In [16]:
process_reviews['image']=process_reviews['image'].fillna('Missing')
In [17]:
process_reviews = process_reviews[process_reviews['reviewText'].notna()]
process_reviews = process_reviews[process_reviews['summary'].notna()]
In [18]:
process_reviews.isnull().sum()
Out[18]:
Concatenating Review Text and Summary¶
In [19]:
process_reviews['reviews']=process_reviews['reviewText']+process_reviews['summary']
process_reviews=process_reviews.drop(['reviewText', 'summary'], axis=1)
process_reviews.head()
Out[19]:
Creating 'sentiment' column¶
In [20]:
process_reviews['overall'].value_counts()
Out[20]:
In [21]:
def f(row):
'''This function returns sentiment value based on the overall ratings from the user'''
if row['overall'] == 3.0:
val = 'Neutral'
elif row['overall'] == 1.0 or row['overall'] == 2.0:
val = 'Negative'
elif row['overall'] == 4.0 or row['overall'] == 5.0:
val = 'Positive'
else:
val = -1
return val
In [22]:
# Applying the function in our new column
process_reviews['sentiment'] = process_reviews.apply(f, axis=1)
process_reviews.head()
Out[22]:
In [23]:
# Sentiment count
process_reviews['sentiment'].value_counts()
Out[23]:
Handling time column¶
In [24]:
# new data frame which has date and year
new = process_reviews["reviewTime"].str.split(",", n = 1, expand = True)
# making separate date column from new data frame
process_reviews["date"]= new[0]
# making separate year column from new data frame
process_reviews["year"]= new[1]
process_reviews=process_reviews.drop(['reviewTime'], axis=1)
process_reviews.head()
Out[24]:
In [25]:
# Splitting the date
new1 = process_reviews["date"].str.split(" ", n = 1, expand = True)
# adding month to the main dataset
process_reviews["month"]= new1[0]
# adding day to the main dataset
process_reviews["day"]= new1[1]
process_reviews=process_reviews.drop(['date'], axis=1)
process_reviews.head()
Out[25]:
5. Text Preprocessing¶
Review Text - Punctuation Cleaning¶
In [28]:
#Removing unnecessary columns
process_reviews=process_reviews.drop(['reviewerName','unixReviewTime'], axis=1)
#Creating a copy
clean_reviews=process_reviews.copy()
In [29]:
def review_cleaning(text):
'''Make text lowercase, remove text in square brackets,remove links,remove punctuation
and remove words containing numbers.'''
text = str(text).lower()
text = re.sub('\[.*?\]', '', text)
text = re.sub('https?://\S+|www\.\S+', '', text)
text = re.sub('<.*?>+', '', text)
text = re.sub('[%s]' % re.escape(string.punctuation), '', text)
text = re.sub('\n', '', text)
text = re.sub('\w*\d\w*', '', text)
return text
In [30]:
process_reviews['reviews'] = process_reviews['reviews'].apply(lambda x:review_cleaning(x))
process_reviews.head()
Out[30]:
Review text-Stop words¶
In [31]:
stop_words= ['yourselves', 'between', 'whom', 'itself', 'is', "she's", 'up', 'herself', 'here', 'your', 'each',
'we', 'he', 'my', "you've", 'having', 'in', 'both', 'for', 'themselves', 'are', 'them', 'other',
'and', 'an', 'during', 'their', 'can', 'yourself', 'she', 'until', 'so', 'these', 'ours', 'above',
'what', 'while', 'have', 're', 'more', 'only', "needn't", 'when', 'just', 'that', 'were', "don't",
'very', 'should', 'any', 'y', 'isn', 'who', 'a', 'they', 'to', 'too', "should've", 'has', 'before',
'into', 'yours', "it's", 'do', 'against', 'on', 'now', 'her', 've', 'd', 'by', 'am', 'from', 'about', 'further', "that'll", "you'd", 'you', 'as', 'how', 'been', 'the', 'or', 'doing', 'such',
'his', 'himself', 'ourselves', 'was', 'through', 'out', 'below', 'own', 'myself', 'theirs',
'me', 'why', 'once', 'him', 'than', 'be', 'most', "you'll", 'same', 'some', 'with', 'few', 'it',
'at', 'after', 'its', 'which', 'there','our', 'this', 'hers', 'being', 'did', 'of', 'had', 'under',
'over','again', 'where', 'those', 'then', "you're", 'i', 'because', 'does', 'all']
In [32]:
process_reviews['reviews'] = process_reviews['reviews'].apply(lambda x: ' '.join([word for word in x.split() if word not in (stop_words)]))
process_reviews.head()
Out[32]:
6. Exploratory Data Analysis¶
In [33]:
# Unique customers and products
print("Total data ")
print("-"*50)
print("\nTotal No of Ratings :",process_reviews.shape[0])
print("Total No of Reviwers :", len(np.unique(process_reviews.reviewerID)))
print("Total No of Products :", len(np.unique(process_reviews.asin)))
In [34]:
# Analysis of rating given by the customer
# Top five customers who have given the most reviews
no_of_rated_products_per_user = process_reviews.groupby(by='reviewerID')['overall'].count().sort_values(ascending=False)
no_of_rated_products_per_user.head()
Out[34]:
Distribution of overall Reviews¶
In [35]:
f, ax = plt.subplots(figsize=(8,5))
ax = process_reviews.overall.value_counts().plot(kind="bar", color = "blue")
ax.set_title("Frequency Distribution of Ratings")
ax.set_xticklabels(df.overall.value_counts().index, rotation = 30)
plt.show()
Distribution of the Target Variable¶
In [36]:
f, ax = plt.subplots(figsize=(8, 6))
ax = sns.countplot(y="sentiment", data=process_reviews, palette="Set1")
ax.set_title("Frequency Distribution of Sentiment Variable")
plt.show()
7. Popularity Based Recommender Analysis¶
In [37]:
#Getting the new dataframe which contains users who has given 50 or more ratings
new_df = process_reviews.groupby("asin").filter(lambda x:x['overall'].count() >=50)
In [38]:
# Average rating of the product
new_df.groupby('asin')['overall'].mean().head()
Out[38]:
In [39]:
# Average rating per product decreasing order
new_df.groupby('asin')['overall'].mean().sort_values(ascending=False).head()
Out[39]:
In [40]:
# Total no of rating for product
new_df.groupby('asin')['overall'].count().sort_values(ascending=False).head()
Out[40]:
In [41]:
# Top 30 popular software products
popular_products = pd.DataFrame(new_df.groupby('asin')['overall'].count())
most_popular = popular_products.sort_values('overall', ascending=False)
most_popular.head(30).plot(kind = "bar")
plt.title('Top 30 popular software products')
plt.xlabel('Product Asin/Product Number')
plt.ylabel('Product Count')
plt.show()
8. Time Series Analysis¶
Year vs Sentiment count¶
In [42]:
process_reviews.groupby(['year','sentiment'])['sentiment'].count().unstack().plot(legend=True)
plt.title('Year and Sentiment count')
plt.xlabel('Year')
plt.ylabel('Sentiment count')
plt.show()
Month vs Sentiment count¶
In [43]:
process_reviews.groupby(['month','sentiment'])['sentiment'].count().unstack().plot(legend=True)
plt.title('Month and Sentiment count')
plt.xlabel('Month')
plt.ylabel('Sentiment count')
plt.show()
Day of Month vs Reviews count¶
In [44]:
#Creating a dataframe
day=pd.DataFrame(process_reviews.groupby('day')['reviews'].count()).reset_index()
day['day']=day['day'].astype('int64')
day.sort_values(by=['day'])
#Plotting the graph
sns.barplot(x="day", y="reviews", data=day)
plt.title('Day vs Reviews count')
plt.xlabel('Day')
plt.ylabel('Reviews count')
plt.show()
Month vs Reviews count¶
In [45]:
#Creating a dataframe
month=pd.DataFrame(process_reviews.groupby('month')['reviews'].count()).reset_index()
month['month']=month['month'].astype('int64')
month.sort_values(by=['month'])
#Plotting the graph
sns.barplot(x="month", y="reviews", data=month)
plt.title('Month vs Reviews count')
plt.xlabel('Month')
plt.ylabel('Reviews count')
plt.show()
Year vs Reviews count¶
In [46]:
#Creating a dataframe
year=pd.DataFrame(process_reviews.groupby('year')['reviews'].count()).reset_index()
year['year']=year['year'].astype('int64')
year.sort_values(by=['year'])
#Plotting the graph
f, ax = plt.subplots(figsize=(15, 6))
sns.barplot(x="year", y="reviews", data=year)
plt.title('Year vs Reviews count')
plt.xlabel('Year')
plt.ylabel('Reviews count'),
plt.show()
9. TextBlob Polarity - Sentiment Classifier¶
In [47]:
process_reviews['polarity'] = process_reviews['reviews'].map(lambda text: TextBlob(text).sentiment.polarity)
Review Length and Word Count¶
In [48]:
process_reviews['review_len'] = process_reviews['reviews'].astype(str).apply(len)
process_reviews['word_count'] = process_reviews['reviews'].apply(lambda x: len(str(x).split()))
In [49]:
process_reviews.head()
Out[49]:
Sentiment polarity distribution¶
In [50]:
process_reviews['polarity'].iplot(
kind='hist',
bins=50,
xTitle='polarity',
linecolor='black',
yTitle='count',
title='Sentiment Polarity Distribution')
Review Rating Distribution¶
In [51]:
process_reviews['overall'].iplot(
kind='hist',
xTitle='rating',
linecolor='black',
yTitle='count',
title='Review Rating Distribution')
Review Text Length Distribution¶
In [52]:
process_reviews['review_len'].iplot(
kind='hist',
bins=100,
xTitle='review length',
linecolor='black',
yTitle='count',
title='Review Text Length Distribution')